Language models often struggle with cross-mode knowledge retrieval -- the ability to access knowledge learned in one format (mode) when queried in another. We demonstrate that models trained on multiple data sources (e.g., Wikipedia and TinyStories) exhibit significantly reduced accuracy when retrieving knowledge in a format different from its original training mode. This paper quantitatively investigates this phenomenon through a controlled study of random token sequence memorization across different modes. We first explore dataset rewriting as a solution, revealing that effective cross-mode retrieval requires prohibitively extensive rewriting efforts that follow a sigmoid-like relationship. As an alternative, we propose CASCADE, a novel pretraining algorithm that uses cascading datasets with varying sequence lengths and computing losses on only the second half of each training sequence to capture knowledge at different scales. Our experiments demonstrate that CASCADE outperforms dataset rewriting approaches, even when compressed into a single model with a unified loss function. This work provides both qualitative evidence of cross-mode retrieval limitations and a practical solution to enhance language models' ability to access knowledge independently of its presentational format.

With his flowing locks, long beard, and worn robes, Jesus is one of the most instantly recognisable figures in the Western world. So it comes as no surprise that his face is also regularly spotted in inanimate objects. This is due to'face pareidolia' - a common brain phenomenon in which a person sees faces in random images or patterns. 'Sometimes we see faces that aren't really there,' explained Robin Kramer, Senior Lecturer in the School of Psychology, at University of Lincoln, in an article for The Conversation. 'You may be looking at the front of a car or a burnt piece of toast when you notice a face-like pattern. 'This is called face pareidolia and is a mistake made by the brain's face detection system.'